Evaluating the usability of a mobile guide: the influence of location, participants and resources

When designing a usability evaluation, choices must be made regarding methods and techniques for data collection and analysis. Mobile guides raise new concerns and challenges to established usability evaluation approaches. Not only are they typically closely related to objects and activities in the user’s immediate surroundings, they are often used while the user is ambulating. This paper presents results from an extensive, multi-method evaluation of a mobile guide designed to support the use of public transport in Melbourne, Australia. In evaluating the guide, we applied four different techniques; field-evaluation, laboratory evaluation, heuristic walkthrough and rapid reflection. This paper describes these four approaches and their respective outcomes, and discusses their relative strengths and weaknesses for evaluating the usability of mobile guides. INTRODUCTION Mobile guides constitute a special class of mobile computer system. Usually mobile guides are closely related to the user’s physical location and objects in the user’s immediate surroundings (e.g. Cheverst et al. 2000, Chincholle et al. 2002, Schmidt-Belz et al. 2002, Reid 2002, Umlauft et al. 2003). Also, they are often used while the user is ambulating, moving from one physical location to another. These properties make the design and evaluation of mobile guides challenging for human-computer interaction researchers and practitioners. The design of mobile guides has received considerable attention over the last decade (see e.g. Abowd et al. 1996, Cheverst et al. 2000, Cheverst et al. 2002, Pospischil et al. 2002, Fithian et al. 2003). When authors consider the design of mobile guides, they also frequently report the results of evaluations. The reported usability evaluations involve the use of a wide range of methods and techniques borrowed from usability research into ‘desk bound’ computers and their use, then adapted to fit the special needs, opportunities and limitations of mobile guides. This includes, for example, formal and informal product presentations combined with questionnaires, expert evaluations (Andrade et al. 2002, Po et al. 2004), controlled laboratory experiments (Bohnenberger et al. 2002, Chincholle et al. 2002, Iacucci et al. 2004) and a variety of use studies in realistic field settings including direct observation of use (Cheverst et al. 2002, Schmidt-Belz and Poslad 2003, Laakso et al. 2003), indirect observation of use (Bornträger et al. 2003), field questionnaires (Rocchi et al. 2003), and longitudinal use studies combined with interviews (Kolari and Virtanen 2003, Iacucci et al. 2004). These evaluations all provide valuable insight into usability and usefulness and typically inform design refinements and/or inspire new design concepts. Such research will, one hopes, result in the development of more useful and usable mobile guides. However, even though evaluations of mobile guides are prevalent, little research has been published on the particular challenges, to usability evaluation, posed by mobile guides; how should we evaluate mobile guides, what methodological challenges do we face, what are the pros and cons of different usability evaluation approaches? Exceptions include, for example, Bornträger and Cheverst (2003) who consider social and technical problems encountered during field evaluations of mobile guide systems, and Kray and Baus (2003) who review and compare nine mobile guide systems and touch upon the methods and techniques that were used in their evaluation. Examining the general literature on mobile HCI does not provide much additional support, with only a few authors considering different usability evaluation methods and techniques for mobile computer systems (see e.g. Brewster 2002, Pirhonen et al. 2002, Kjeldskov and Skov 2003, Kjeldskov and Stage 2004). As a result of our reluctance to ‘evaluate evaluation’, that is to understand how the utility of the techniques in our usability toolkit respond to the challenge of mobile guide evaluation, no agreed upon set of